The Haskell package static-bytes was spun out of the pantry package. Ilias Tsitsimpis reported that static-bytes-0.1.0 did not work as intended on big-endian machine architectures, specifically IBM’s s390x. A big-endian architecture stores the most significant byte of a multi-byte word at the lowest memory address. A little-endian architecture does the opposite. x86_64 is little-endian and AArch64 is little-endian on most machines.
DynamicBytes and StaticBytes
The static-bytes-0.1.0 package makes use of two type classes DynamicBytes and StaticBytes. The functions promised by the classes are not exported. The types provided by the package (Bytes8 to Bytes128) are instances of StaticBytes. Types that provide sequences of (8-bit) bytes are instances of DynamicBytes (currently, Data.ByteString.ByteString, Data.Vector.Primitive.Vector word8, Rio.Vector.Storable.Vector word8 – a re-export from Data.Vector.Storable – and Rio.Vector.Unboxed.Vector word8 – a re-export from Data.Vector.Unboxed).
The type classes are used in the constraints of conversion functions. The most tolerant one to convert from dynamic to static is toStaticPadTruncate:
|
1 |
toStaticPadTruncate :: (DynamicBytes dbytes, StaticBytes sbytes) => dbytes -> sbytes |
fromStatic converts from static to dynamic:
|
1 |
fromStatic :: forall dbytes sbytes. (DynamicBytes dbytes, StaticBytes sbytes) => sbytes -> dbytes |
Let’s look at Data.ByteString.ByteString and Bytes16 as an example.
ByteString and Bytes16
The implementation of toStaticPadTruncate is:
|
1 2 3 4 5 |
toStaticPadTruncate :: (DynamicBytes dbytes, StaticBytes sbytes) => dbytes -> sbytes toStaticPadTruncate dbytes = unsafePerformIO (withPeekD dbytes (usePeekS 0)) |
withPeekD is promised by the DynamicBytes class. For the ByteString instance, it is:
|
1 2 3 4 |
import qualified Data.ByteString.Internal as B withPeekD :: ByteString -> ((Int -> IO Word64) -> IO sbytes) -> IO sbytes withPeekD = withPeekForeign . B.toForeignPtr |
toForeignPtr :: ByteString -> (ForeignPtr Word8, Int, Int) deconstructs a ForeignPtr from a ByteString. The first Int is the offset (which is 0) and the second Int is the length of the string of bytes in bytes.
withPeekForeign is:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
withPeekForeign :: (ForeignPtr word8, Int, Int) -> ((Int -> IO Word64) -> IO sbytes) -> IO sbytes withPeekForeign (fptr, off, len) inner = withForeignPtr fptr $ \ptr -> do let f off' | off' >= len = pure 0 | off' + 8 > len = do let loop w64 i | off' + i >= len = pure w64 | otherwise = do w8 :: Word8 <- peekByteOff ptr (off + off' + i) let w64' = shiftL (fromIntegral w8) (i * 8) .|. w64 loop w64' (i + 1) loop 0 0 | otherwise = peekByteOff ptr (off + off') inner f |
inner, given a function that takes an Int (an offset) and yields an action providing a Word64, yields an action that provides a value of type b (in this instance sbytes). In this case, the function given to inner is f. f puts up to 7 bytes into a Word64 on the assumption that the first bytes are the least significant or it puts 8 bytes directly into a Word64. It seems to me that, on a big-endian machine, the latter will treat the first bytes as the most significant.
In this case, inner is usePeekS 0. usePeekS is promised by the StaticBytes class. For the Bytes16 instance, it is:
|
1 2 |
usePeekS :: Int -> (Int -> IO Word64) -> IO Bytes16 usePeekS off f = Bytes16 <$> usePeekS off f <*> usePeekS (off + 8) f |
The data constructor of Bytes16 is not exposed, but it is Bytes16 !Bytes8 !Bytes8. Bytes8 is simply a newtype for Word64. The first field is filled first and then the second field is filled.
For the Bytes8 instance, usePeekS is:
|
1 2 |
usePeekS :: Int -> (Int -> IO Word64) -> IO Bytes8 usePeekS off f = Bytes8 <$> f off |
The implementation of fromStatic is:
|
1 2 3 4 5 |
fromStatic :: forall dbytes sbytes. (DynamicBytes dbytes, StaticBytes sbytes) => sbytes -> dbytes fromStatic = fromWordsD (lengthS (Nothing :: Maybe sbytes)) . ($ []) . toWordsS |
The first function to be applied in the definition of fromStatic, toWordsS, is promised by the StaticBytes class. For the Bytes8 instance it is:
|
1 2 |
toWordsS :: Bytes8 -> [Word64] -> [Word64] toWordsS (Bytes8 w) = (w:) |
and for the Bytes16 instance it is:
|
1 2 |
toWordsS :: Bytes16 -> [Word64] -> [Word64] toWordsS (Bytes16 b1 b2) = toWordsS b1 . toWordsS b2 |
The second field is added to the list of Word64 and then the first field is added. So, toWordS b is adding to the head of a list of Word64 and the least signficant Word64 is added last.
The next function to be applied, ($ []) :: ([a] -> b) - > b, starts the list of Word64 with an empty list.
The final function to be applied in the definition of fromStatic, fromWordsD (lengthS (Nothing :: Maybe sbytes)), is promised (fromWordsD) by the DynamicBytes class. For the ByteString instance it is:
|
1 2 3 4 5 6 |
fromWordsD :: Int -- ^ The length of the string of bytes -> [Word64] -> ByteString fromWordsD = fromWordsForeign (`B.fromForeignPtr` 0) |
The implementation of fromWordsForeign is:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
fromWordsForeign :: (ForeignPtr a -> Int -> b) -> Int -> [Word64] -> b fromWordsForeign wrapper len words0 = unsafePerformIO $ do fptr <- B.mallocByteString len withForeignPtr fptr $ \ptr -> do let loop _ [] = pure () loop off (w:ws) = do pokeElemOff (castPtr ptr) off w loop (off + 1) ws loop 0 words0 pure $ wrapper fptr len |
I assume that the behaviour of pokeElemOff :: Ptr Word64 -> Int -> Word64 ->IO () depends on the endianess of the machine architecture.
The fromWordsD of the ByteString instance is also used to implement the Bytes8 instance of Show, as follows:
|
1 2 |
instance Show Bytes8 where show (Bytes8 w) = show (fromWordsD 8 [w] :: B.ByteString) |
Enforcing endianess
The fix was to make use of helper functions to enforce the endianess of Word64 values (with names inspired by the names of similar functions provided by the cpu package):
|
1 2 3 4 5 6 7 8 9 10 11 |
import GHC.ByteOrder ( ByteOrder (..), targetByteOrder ) -- | Convert a 64 bit value in CPU endianess to little endian. toLE64 :: Word64 -> Word64 toLE64 = case targetByteOrder of BigEndian -> byteSwap64 LittleEndian -> id -- | Convert a little endian 64 bit value to CPU endianess. fromLE64 :: Word64 -> Word64 fromLE64 = toLE64 |
So, in withPeekForeign we have:
|
1 |
otherwise = toLE64 <$> peekByteOff ptr (off + off') |
and in fromWordsForeign we have:
|
1 |
pokeElemOff (castPtr ptr) off (fromLE64 w) |